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Abstract 

Although simulation represents a major advance in the understanding of problems in 
complex systems, the field currently does not has standards in place that would guide the 
reporting of the data underlying each model, the process for model validation. This lack of 
common platforms significantly decreases the attractiveness of the field in that comparison 
across models is difficult, and peer evaluation of whether models should be trusted and 
used in practice is severely limited. This article reports on a scries of concepts that could 
serve as the basis for an initial discussion regarding potential platforms wiht the simulation 
community. The document covers a proposed platform for research question formulation, 
literature review, data collection, model analysis, and manuscript writing. Considerations 
are then made to counter the prediction that raising quality levels would lead to a decreasing 
rate in expansion for the field. 
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1 Introduction 



Simulation represents a major analytical advance in a number of scientific fields, being used 
for both prediction as well as understanding of problems in systems demonstrating complex 
behavior pp. Used in multiple fields, from economics to medicine, simulation frequently generates 
substantial insights regarding systems that otherwise would be only partially understood through 
piecemeal analytical approaches that are incapable of apprehending the multiple interactions, 
feedback loops, and unintended consequences associated with complex environments. Despite its 
significant methodological achievements, simulation models are often plagued by a development 
methodology that is unclear, based on data sources that are undefined, and drawing relations 
to the real world that are questionable. 

To put the simulation of complex systems in perspective, a contrast should be established 
with previous methods and their limitations. These previous methods include classical tech- 
niques of experimentation that can only control for a few variables at a time, deterministic solu- 
tions that require an exact analytical solution, and classical statistical models that are primarily 
bound to quantitative data sets. In contrast, simulation allows for the inclusion of quantitative 
and qualitative data, complex interactions among multiple variables, and model manipulation 
in a variety of ways that allow for prediction, better understanding of the model in relation 
to the problem being studied, and the analysis of multiple hypothetical interventions. All of 
these activities are, however, pending not only on how the model was originally constructed 
(i.e., which sources of data and information were considered in its formulation), but also how 
the model has been validated (i.e., compared to problems in the environment where the problem 
originally occurred). Currently, a variety of models in the literature lack any use of platforms 
to streamlined how these models are built and validated [2]. 

The objective of this proposal is therefore to review the literature and propose an initial 
set of platforms for the creation of simulation models, common to System Dynamics (SD). 
These platforms are divided into research question formulation, literature review, data collection, 
modeling process, and manuscript reporting, each one covered in the following sections. 

2 Research Question Formulation 

Based on a compilation of previous references from SD literature f3], we propose that the 
process of research question formulation be composed of three main section. Namely, these 
sections are (1) Hypothesized conclusion, (2) Variable selection, and (3) Mock results. 

2.1 Problem definition through a "hypothesized conclusion" 

Hypothesized conclusions follow the Feinsteinian tradition [3] of stating the supposed con- 
clusion of a study in order to extract the real intent of the researcher stating the problem to be 
resolved. 

According to Feinstein, whenever a researcher attempts to formulate the problem to be solved 
in terms of hypothesis or some other vague statement, their ultimate purpose and belief of what 
they would like the model to ultimately solve or display is hidden. Another important aspect 
in this process is that the model has to be based on a problem rather than on an attempt to 
mimic a system. Because systems are complex and can be perceived from multiple perspectives 
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[5], a system could be modeled in a number of different ways depending on the problem that 
the modeler is attempting to approach. 

2.2 Variable selection 

Variable selection is important in that it will assist in the creation of an outline of the SD 
structure. Variable selection is primarily determined by a combination of information from 
the literature as well as quantiative and qualitative data, which are treated in more detail 
in subsequent sections (sections 3 and 4, respectively). Finally, variable selection is closely 
related to the hypothesized conclusion, in that it should contain all variables required to address 
the problem, but probably no additional variables in order to decrease model complexity and 
unnecessary work. 

The variable selection process involves the compilation of factors that are believed to be 
implicated in the problem to be modeled. For SD models, this list can later be organized 
in a causal loop diagram demonstrating the feedback loops involved in their interaction as 
demonstrated in the "Standard Method" outlined by Lyneis [4j. 



2.3 Mock results 

Mock results are important to align the problem statement outlined in the hypothesized 
conclusion with the list of variables. Briefly, when the mock results do not clearly present an 
answer to the stated problem or when it is based on variables that have not been previously 
listed, the research question formulation process is inconsistent and should be revised. Mock 
results take a number of forms outlined in the following sections. 

2.3.1 Reference modes 

Reference modes constitute a graphical representation of what modelers believe the system 
response will be over time [4]. This representation usually contains best and worst case scenarios. 
For example, in the case of a citation network, a high-impact article is expected to obey a power- 
law up to the point when that citation ages, when its probability of being cited starts declining 
(Figure [TJ . 

2.3.2 Selection of archetypes and template models 

Once the reference modes are stated, the next step is to search model libraries for pre- 
existing models or archetypes that can provide structural templates that mimic the expected 
system behavior. This process, sometimes called dynamic hypothesis, represents the association 
between model structure and expected system behavior. This phase can be compared to the 
sequence involved in statistical modeling where a family of distributions might be initially chosen 
to model a given process, the fit of this distribution family to the actual data being later 
validated. Similar to the statistical modeling process, initial assumptions can later be replaced, 
and this is often the case with archetypes. In sum, they function as initial templates rather than 
determine how the system should be modeled. 

SD archetypes have been portrayed as the structural basis determining common types of 
system behavior encountered in complex models [S];[7]; [5] ; [2] ; [313] • When some of the expected 
behavior determined by reference modes is mapped to a certain archetype, that archetype usually 
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provides insight into the model structure. This statement should not mean that the structure of 
the model should be driven by the archetype, but simply that archetypes can assist in provid- 
ing ideas regarding potential mechanisms that influence the observed behavior. An additional 
resource is the concept of molecules [TTJ , initially developed by Jim Hines [12] . Molecules consti- 
tute the building blocks of SD models or small templates that can be assembled and customized 
to create larger models. 

The models can then be used, in modules or as a whole, as the basis for modeling the new 
problem. Libraries for SD models are somewhat limited, although the public library from Tom 
Fiddaman [13] is available, also including personal libraries such as Jim Thompson's personal 
library for healthcare models (Jim Thompson, personal communication). 

2.3.3 Knowledge engineering and model libraries 

Although existing libraries represent an important resource, the navigation through their 
content presents significant limitations. Libraries usually label and describe their models focus- 
ing on their main content area (e.g., an emergency room, a fishery model), whereas modelers 
searching for model templates are usually interested in models with certain behaviors or struc- 
tural characteristics. A potential solution for this problem would be the development of a 
simple computational ontology |14] allowing modelers to perform rich semantic search strategies 
to locate the best possible model to fit their purposes. This ontology would contain elements 
describing four main model characteristics, namely, content area, structure, behavior, and model 
quality. While in this article we will not focus on content area, likely organized around a folkson- 
omy j!5| , structure and behavior (requiring a group of experts proposing an initial classification 
validated through the application to existing libraries) , the issue of model quality is central and 
will therefore be more explored in this article. 

A wide variation in model quality variation is pervasive in much of the computer simulation 
literature. Two essential areas are frequently unclear whenever a model is built, analyzed and 
reported: (1) Data sources used to build the model and (2) How the model is validated or, 
stated in a different manner, how its behavior relates to the environment where the problem is 
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located. These two areas are addressed in the following sections. 

Data sources for baseline simulation models 

The proposed classification for data sources in baseline models assumes that the data should 
be classified in three categories: Quantitative, qualitative, and personal experience (Table 1). 



• Qt - quantitative data 

o no quantitative data 

: 1. tables or graphics extracted from secondary sources 
o 2. primary raw data from referenced source 

• Ql - qualitative data - qualitative interviews or observations, literature review 

o 0. no qualitative data 

o 1. informal interview or conversations with stakeholder, quotes from existing 
documentation 

o 2. structured interviews - including formal recording, transcripts and coding 

• P - personal experience - stakeholders involved in the process (experts, leaders, 
users, etc) 

o no data relying solely on personal experience 
o 1 data relying on personal experience 

• Example 

o Model with 3 loops 

■ each loop classified depending on the underlying of data 

■ example with list of loops and corresponding QtQIP classification : 

■ Rl = 120 

■ Bl = 110 

■ R2 = 001 

■ overall, with max and min boundaries = 0-1 0-2 0-1 



Figure 2: Data source classification for baseline models 

For SD models each loop is classified separatel. This proposed model should undergo exten- 
sive validation using existing libraries before it can be presented as an initial working version. Of 
importance, this classification does not stand for a quality judgment or criteria for selecting one 
model over another. It also has no implied assumptions on whether qualitative data might be 
superior to quantitative or vice- versa. It simply clarifies where exactly the data and information 
underlying the model comes from, thus allowing readers to judge for themselves on whether they 
should or not trust model results. 

The second point where further quality assurance for the model library is required is regard- 
ing model validation or how the baseline model corresponds to the behavior presented in the 
environment where it is located. A simplified classification for validation is proposed in Table 2. 

Additional comments on model validation will be made in section 5.2. 

Model library governance 

Governance for this library will include minimal quality criteria for model inclusion as well 
as incentives for participation. The latter could include, for example, the provision of references 
for contributed models that can then be used for citation in articles, curricula, funding proposals, 
among others. This incentive framework has been successfully implemented in other projects 
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• Qt - quantitative data 

□ predictions not compared to any quantitative data 

o l predictions compared to tables or graphics extracted from secondary sources 

□ 2 predictions fitted against primary raw data from referenced source 

• Ql - qualitative validation - only evaluates shape or overall pattern of the behavior 

o predictions not compared to any qualitative data 

□ 1 predictions compared to informal interview or conversations with stakeholders 
on what could actually happen 

o 2 predictions compared to interviews on what actually happened 

• P - personal validation - face validity regarding the CLD 

o no data relying solely on personal prediction 

□ 1 data relying on personal prediction of what could potentially happen 
» Example using QtQIP 

o Policy prediction would be classified as a whole as 110 



Figure 3: Validation for baseline and policy models 

involving data and model contribution such as Dataverse j!6j . 

2.3.4 Momentum policies 

Momentum policy [1] is a term used within the SD community to represent proposed solutions 
to the problem being modeled, based on insights obtained from the baseline model. The potential 
solutions are implemented in the baseline model, observing whether its behavior changes in the 
direction expected toward the improvement of the proposed problem. Similar to the issue of 
data quality for the baseline model, the description of momentum policies in the simulation 
literature is frequently unclear, and so we propose that the validation of momentum policies be 
performed according to the same classification proposed in Table 2, pending further refinement 
by peers in the SD communities. 

3 Literature review 

3.1 Content area 

Protocols on streamlined methods to conduct literature review have been prepared by the 
Research on Research Group at Duke University on how to conduct a literature review |17j . 
These platforms are related to the research question formulation as well as data collection for 
modeling [18J. Briefly, the method of Literature Matrices [19] is proposed, with articles being 
grouped in clusters associated with loops for SD. 

3.2 Simulation modeling 

A literature review for SD modeling is facilitated by the search of standard sources of in- 
formation. These include the bibliographic compiled and constantly updated by the System 
Dynamics Society [20], as well as list of journals and books on systems modeling currently in 
preparation by the Research on Research (RoR) group at Duke University. In combination with 
the model library described in section 2.3.3 and mentorship during distance research coaching 
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programs, these resources are expected to improve the overall quality and productivity of the 
models according to a platform agreed upon by the simulation community. 

4 Data collection 

4.1 Qualitative data 

Methods for literature review as a source of qualitative information for simulation models 
were addressed in section 3. Also, previous literature has been written on optimal methods to 
conduct qualitative studies to interview stakeholders (experts, users, among others) providing 
information that can contribute to the simulation model. This literature has focused on both 
qualitative methods in general |21|:|22j: |23j as well as specific literature for simulation modeling 
|24j ; |25j . Rather than claiming for the superiority of one method vs. another, a platform will 
simply offer a bibliography on the topic so that modelers can cite their method of choice. More 
important than pre-judging which method is superior in each situation, appropriate documen- 
tation describing which method was used is essential so that readers can judge whether they 
should believe or not in the inferences made from the model based on the quality of the data. In 
situations where the literature is either non-existing or of poor quality, a Delphi method [26] can 
be used to obtain expert opinion as the basis for the model. Almost invariably Delphi methods 
will be the most commonly used methods to define the overall structure of the model as this 
information is not available from the literature unless previous SD models might already be 
available from the literature. In conducting a Delphi method, it is important that its methods 
be reproducible. A dynamic protocol has been developed by the RoR Group that addresses 
reproducibility within Delphi \27\ . 

4.2 Quantitative data 

The documentation for quantitative methods that might serve as the basis for a model is less 
well-defined, but we believe that transparency and potential reproducibility continue to be key 
parameters to be considered. Therefore, previous models advocating for making data openly 
available along with the corresponding scripts that gave rise to the analysis used in the results 
fed into the model should be preferred. As examples, we cite the sweave [28] and zelig [29] 
libraries for the R statistical language [30] as optimized protocols to accomplish this level of 
transparency. 

5 Modeling process 

When it comes to the modeling process itself, a platform should be in place that would 
substantially streamline this process, ultimately improving their quality and productivity. We 
only focus on recommendations regarding modularity, distributed versioning, and connection 
across modeling software packages. 
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5.1 Analytical solution / SD 

5.1.1 Modularity and distributed versioning in model development 

One of the central characteristics of successful distributed software projects is the ability to 
create modules that can be executed by individual programmers in a streamlined fashion [31] . 
Because systems modeling usually requires the expertise of multiple specialists from a variety 
of disciplines, and the criticisms of other modelers on how the model should be structured, 
validated, and reported, model modularity is a key for the expansion of the field. Modularity 
in systems modeling can be accomplished by proper documentation of additional structural, 
behavioral, or validation tasks that should be accomplished, accompanied by the creation of a 
reliable and easy to use distributed versioning system. 

At this point, our group is evaluating the use of Git [32] since it seem to be more amenable 
to the distributed nature of our network [33] contributing to simulation modeling in comparison 
with previous tools such as subversion [M]. Ideally, these repository would be made publicly 
available, so that modelers can download files from regular web pages while those actively 
working on modifying the model can upload modifications directly to Git. 

5.1.2 Connection across modeling software packages 

One of the major limitations in the field is the lack of widespread interoperability across 
different modeling software packages. Ideally, one would like to see the implementation of 
platforms that allow for at least the connection among three major classes of software packages, 
including statistical packages (such as the open source package R) [30]; SD-specific packages 
(such as Vensim |35j) , Stella/Ithink [36J , and Powersim [37 . 

5.2 Validation 

An extensive literature has been created regarding the multiple challenges associated with 
the validation (or impossibility of validation) of simulation models using SD. The literature 
has plenty of personal opinions, many of them related to the local experience of the modeler 
with a restricted set of modeling problems while disregarding problems that are in fields that 
are foreign to them. Rather than arguing for one approach rather than another, the simple 
platform displayed in Table 2 (pending modifications and further testing) is proposed, pending 
further input and refinement by the SD community. 

6 Manuscript writing 

While platforms in the creation and validation of simulation models are essential, they will 
realize their full value if they are properly documented in the final article or report. This section, 
which proposes a platform for a traditional research paper, relies on information made available 
by the RoR Group [T7] from the Duke University Medical Center. It includes a classical IMReD 
framework, including Introduction, Methods, Results and Discussion. For each section, a general 
discussion of its contents is provided. 
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6.1 Introduction 



6.1.1 Significance 

The first paragraph should state why the topic to be treated in the paper is important. This 
statement has to convince readers that they should keep on reading the manuscript given its 
relevance. 

6.1.2 Information gap 

This section should state that, despite being relevant as made clear in the previous section, 
there is an important information gap in the field. It will be clear in the last section of the 
introduction that this gap is not any gap, but the gap that will be evaluated by this article. In 
other words, this section should sell the idea of the importance of your paper to the reader. 

6.1.3 Literature review in support of information gap 

This section reviews the literature on the gap border while pointing to information gap in 
our current knowledge that will be solved by the article paper. The role of this section is not 
to review all the knowledge in the field - that is left for review articles. Instead, the role of 
this section is to provide readers with the background information necessary to understand the 
article while constantly emphasizing that no previous article has evaluated the issue that this 
article will approach. Because the gap is a place where no previous information exists, novice 
researchers tend to make two mistakes. First, review the literature surrounding the gap while 
focusing on that literature. The second common mistake is to attempt to run a literature search 
focusing only on the gap. Although a search attempting to find information on the gap should 
certainly be the first step, researchers should not find anything on the gap, otherwise it would 
not be a gap. A gap, by definition, is an area where no previous information is available. In 
summary, the literature search to support the existence of the gap is a search focused on the 
surroundings of the gap, while focusing on making that gap evident to the reader. In other words, 
researchers should search for the gap and expect to find nothing, and then start expanding their 
search to the areas surrounding the gap until they find some literature. This "border" is then 
reviewed and criticized for not having yet approached the gap. 

6.1.4 Objective and hypotheses 

The last section concludes the introduction by stating the main objective of the paper and 
respective hypotheses. Of importance, the objective fills the information gap previously stated 
in the second section of the introduction. 

6.2 Methods 

6.2.1 Problem definition 

This section describes the problem to be solved by the model, including how the baseline 
model can be used to predict the system behavior through policy models. This section of the 
article follows the recommendations of section 2.1. 
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6.2.2 Rationale for choice of modeling method 

The model should mimic the conditions of the environment where the problem is situated, 
explaining why its advantages over other possible research methodologies such as classical exper- 
imental methods, deterministic solutions, and classical statistical models. The choice of model 
(SD) should be described along with a brief explanation of what that modeling option entails. 

6.2.3 Variable selection 

This section describes each of the variables used in the baseline as well as policy models. For 
more details, see section 2.2. 

6.2.4 Reference modes 

This section describes the reference modes. For more details see section 2.3.1. 

6.2.5 Templates and previous examples from model library 

This section describes previous templates or archetypes that might have been used as the 
basis to address the problem in the current model. For details, see section 2.3.2. 

6.2.6 Data sources for baseline and policy models 

In this section an explanation of the literature sources (see section 3), sources and methods 
of qualitative data (see section 4.1), and sources and methods of quantitative modeling process 
(see section 4.2) are described. This section includes the classification proposed in Table 1. 

6.2.7 Measures of model validation 

This section determines how the baseline model was comparable to the qualitative and 
quantitative sources of data as well as which criteria were used to consider the baseline model 
adequately reproducing the problem in the system being modeled. This includes a description 
of momentum policies (see section 2.3.4). This section should include the classification proposed 
in Table 2, pending further refinement by peers in the SD community. 

6.2.8 Software 

This section will list software packages and programming languages used in the modeling 
process. 

6.3 Results 

6.3.1 Baseline model 

This section presents the overall description of the baseline model. For SD this would include 
a description of each of the loops represented by causal loop as well as stock and flow diagrams. 
Source code should be provided in the appendix so that the model can be reproduced. This 
description is followed by graphics and tables describing model performance metrics at baseline. 
A description of how the model compares to the problem in the original environment should 
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be provided, preferably with graphics and tables that allow the reader to compare and judge 
whether model and problem in the original environment are indeed comparable in their behavior 
and/or predictions. 

6.3.2 Policy models 

This section presents modifications to the baseline model in terms of all variables and struc- 
ture to the elements of SD. Graphics and tables should be presented comparing multiple policy 
models, emphasizing in which situations each of the alternative models might be better than 
others. Information on how these policy models might compare to what would happen for 
problems in the environment with some of these modifications should be provided. Examples 
would include partial implementations in similar environments, slightly different implementa- 
tion in similar environments, or any other data or information that might enhance the readers' 
ability to judge whether the policy results in the model are indeed likely to result in similar 
modifications in the environment where the problem is located. 

6.4 Discussion 

6.4.1 Strengths and summary of main findings 

In this section the main strengths of the model are stated, while summarizing the main 
results. If possible, it should be stated how the model is unique compared to the remaining 
literature. The number of main findings is usually restricted to a maximum of four, with a short 
statement describing each one. 

6.4.2 Discussion of each main model results 

This section compares each main result of the model (usually three to four) with the liter- 
ature. First, it should discuss the findings that agree with previous studies, later approaching 
the model results that disagree with the literature. Attempt to provide explanations on why 
the disagreement occurred. Disagreements are usually related to a different research methodol- 
ogy and/or a different patient characteristics. Both qualitative and quantitative aspects of the 
results should be discussed. This section differs from previous portions of the report in that it 
allows for more freedom of opinion about interpretation. That said, common sense is always 
advisable. 

6.4.3 Study limitations 

This section points to the main limitations of your study and what you have done to min- 
imize those limitations (if nothing can be done to minimize them, explain how the limitation 
didn't have much of an influence on the results) . It should also mention analyses that were not 
performed in the present study and that should be approached in subsequent studies in order 
to complement the results. 

6.4.4 Conclusions 

This section states the main conclusions based on model findings, including policy implica- 
tions, potential changes in practice, and future research opportunities. 
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7 Some concerns: Would a platform limit further expansion of 
the field or even impose limits to its creativity? 

One of the common concerns regarding the implementation of platforms in a scientific field 
is that they will limit the expansion of the field since the bar will be set too high for newcomers. 
Since the inflow of newcomers would be lower, the number of members would decrease over 
time. We would argue that this statement oversees that the what attracts newcomers is not 
that the field has scientific rules that welcome a wide variety of quality levels in modeling 
studies. Instead, what would attract newcomers is if suggested platforms were in place so that 
results can be compared across different models, ultimately leading the field might to accumulate 
information and later be taken into new, more sophisticated directions. Although not having 
any empirical evidence to demonstrate this statement, the concern raised by those concerned 
with setting a platform seems follows along the SD archetype called "drifting goals" [38], in that 
not adhering to a platform undermines the intended balance to increase the size and quality of 
the overall simulation community. 

8 Conclusion 

This article outlines some initial concepts that can be used as a springboard toward the 
creation of a standardized platform for the design, data instantiation, and reporting of System 
Dynamics models. We do not cover standards for model dissemination, although we believe that 
methods such as Mediated Modeling [39jare equally important to ensure that SD models can be 
used to modify practice rather than simply serve as a theoretical exercise. We hope that some 
of these ideas can elicit discussion among SD modelers. 
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